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ABSTRACT 


Tutorial dialogue is a highly effective way to support stu- 
dent learning. It is widely recognized that tutor dialogue 
moves can significantly influence learning outcomes, but the 
ways in which tutor moves, student affective response, and 
outcomes are related remains an open question. This paper 
presents an analysis of student affective response, as evi- 
denced by multimodal data streams, immediately following 
tutor questions. The findings suggest that students’ affect 
immediately following tutor questions is highly predictive 
of end-of-session self-reported engagement and frustration. 
Notably, facial action units which have been associated with 
emotional states such as embarrassment, disgust, and happi- 
ness appear to play important roles in students’ expressions 
of frustration and engagement during learning. This line of 
investigation will aid in the development of a deeper under- 
standing of the relationships between tutorial dialogue and 
student affect during learning. 
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1. INTRODUCTION 


Tutorial dialogue provides rich, natural language adap- 
tation to students during learning. An understanding has 
emerged about the role of interactivity in tutorial dialogue 
[40, 6] and on dialogue strategies for most effectively sup- 
porting students in task-oriented tutorial dialogues [29, 10]. 
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However, a pressing issue is developing an understanding 
of how specific tutor dialogue moves impact students’ affect, 
and in turn, what influence students’ affective responses may 
have on outcomes. 

The need for modeling affect during learning is widely rec- 
ognized. Research has shown that suites of affect detectors 
from sensors and log files can perform well but that there 
are trade-offs depending on the goals of the affect detection 
modules [22, 33]. Affect detectors have been investigated for 
a wide variety of affective states including confidence, ex- 
citement, frustration, and interest [41], and within tutorial 
dialogue, for uncertainty [11]. There have also been great 
strides in sensor-free affect detection which relies primarily 
on log files [2]. This approach has shown promise during 
cognitive tutoring [9] and for distinguishing frustration and 
confusion [27]. 

Out of all of the affective phenomena that have been 
examined during learning, two affective states are frustra- 
tion and engagement. These states have been examined 
in fine-grained analyses as tutoring unfolds, and also as 
outcome measures regarding students’ perceptions of the 
success of the tutoring session. Engagement and frustra- 
tion have been predicted at above-chance levels using facial 
expression-based affect detection even without the presence 
of interactive events during text or diagram comprehension 
[5]. Engagement and frustration have also been predicted 
with nonverbal behaviors, including facial expression, af- 
ter student task events during problem solving [16]. In a 
compelling development, emerging evidence shows that fine- 
grained affective events can have long-lasting relationships 
with outcomes that may be far removed from those affective 
events [36]. 

This paper advances the understanding of student emo- 
tions in learning by examining students’ fine-grained affec- 
tive responses to tutor questions during tutorial dialogue. It 
investigates the hypothesis that students’ affective responses 
immediately following tutor questions are related to self- 
reported frustration and engagement at the end of the ses- 
sion. The results indicate that several key facial expression 
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features immediately following two different types of tutor 
questions are highly predictive of end-of-session self-reported 
engagement and frustration. This line of investigation rep- 
resents a step forward in understanding the affective impact 
of tutorial strategies. 


2. RELATED WORK 


Tutorial dialogue researchers have long studied what hu- 
man tutors naturally do: how strategies differ between ex- 
perts and novice tutors [12] whether Socratic or didactic ap- 
proaches are most effective [35] and how tutors scaffold and 
fade support during problem solving [4], among others. The 
impact of particular tutorial dialogue moves has been the 
focus of significant attention, with findings indicating that 
positive and negative feedback have different impact based 
on students’ self-efficacy level [3], that bottom-out directives 
are not conducive to learning [29], and that adapting to stu- 
dent uncertainty improves the effectiveness of tutorial dia- 
logue [10]. However, this paper examines a different aspect 
of these tutorial dialogue moves that is critical in learning: 
students’ affective response as expressed on the face and as 
embodied in gestures. 

Multimodal features such as dialogue, facial expression, 
posture, and task actions have been used to predict affective 
states, such as boredom, confusion, excitement, and frustra- 
tion, as those states occur during learning [23, 8, 7]. More- 
over, multimodal features such as facial expression and ges- 
tures can significantly predict frustration and engagement 
reported at the end of tutoring sessions [17], and some differ- 
ences have emerged in the extent to which upper and lower 
facial expression features are associated with these outcomes 
[15]. This previous work on utilizing multimodal features 
for predicting frustration and engagement during human- 
human tutoring has emphasized the important role that tu- 
tor dialogue moves play in affective outcomes. Other factors, 
such as student personality profile, can also contribute signif- 
icantly to predicting these outcomes [39]. The present work 
examines moment-by-moment affect as evidenced by mul- 
timodal traces, and then analyzes the relationship between 
these multimodal behaviors and the outcomes of frustration 
and engagement as reported by students after the tutoring 
session. 


3. STUDY DATA 


The present analysis investigates the multimodal behavy- 
ior of students during a computer-mediated tutorial session 
in introductory computer science, and specifically in Java 
programming [18, 30]. The tutorial interface, shown in Fig- 
ure 1, is divided into four panes: the task description, the 
student’s Java source code, the compilation and execution 
output of the program, and the textual dialogue messages 
between the tutor and the student. The tutor’s interactions 
with the environment were constrained to progression be- 
tween tasks and sending textual messages to the student. 


Students (N = 67) were university students in the United 
States enrolled in an introductory engineering course, with 
an average age of 18.5 years (s = 1.5 years), whereas the hu- 
man tutors (N = 5) were primarily graduate students with 
previous experience in tutoring or teaching introductory pro- 
gramming. The behavior of the student was collected using 
a set of multimodal sensors, as shown in Figure 2, including 


a Kinect depth sensor, an integrated webcam, and a skin 
conductance bracelet. The following subsections detail the 
modalities appearing significant in the present analysis. 

Each student participated in six 40-minute sessions over 
the course of four weeks; however, the present analysis only 
examines data from the first lesson. Before and after each 
lesson, students completed a content-based pretest and iden- 
tical posttest; the tutoring sessions were found to be signif- 
icantly effective in facilitating learning gains (p < 0.0001). 
In addition to the posttest, students also completed a post- 
survey, including the NASA-TLX workload survey [20] and 
the User Engagement Survey [32]. The present analysis in- 
vestigates self-reported frustration, taken from the Frustra- 
tion Level item of the NASA-TLX workload survey, and en- 
gagement, taken as an average of three sub-scales of the User 
Engagement Survey: Focused Attention (perception of time 
passing), Felt Involvement (perception of involvement with 
the session), and Endurability (perception of the activity as 
worthwhile). 


3.1 Task Event and Dialogue Features 


During the tutoring session, the interface described above 
logged tutor and student dialogue messages, student typing 
in the code window, and student progress through the task. 
No turn-taking measures were enforced in the dialogue: stu- 
dents and tutors could send messages to the other at any 
point. All exchanged messages were automatically tagged by 
a J48 decision tree classifier [37] with a dialogue act annota- 
tion scheme created for task-oriented tutorial dialogue that 
differentiates tutor questions, feedback, and hints, among 
other dialogue moves [38]. In that work, the Cohen’s kappa 
between two human annotators was 0.87 and the Cohen’s 
kappa between human and the J48 decision tree classifier 
was 0.786. 

The analysis presented here focuses on two types of tutor 
dialogue moves: inference questions and evaluative ques- 
tions. (Although other question types were investigated, 
student reactions to these were not found to have significant 
predictive power.) Inference questions require the formation 
of an action plan or reasoning about existing content knowl- 
edge. For example, ‘How do you think this problem can be 
solved?’, or ‘How can you fix this error?’ are considered to 
be inference questions. On the other hand, evaluative ques- 
tions aim to evaluate the student’s belief in his or her own 
understanding of the material, e.g., ‘Does that make sense 
so far?’, or ‘Do you understand?’ (see Figure 4). 

Previous work has suggested that questions can stimu- 
late cognitive disequilibrium in a student [34], which is often 
considered to be a critical step in knowledge acquisition [13]. 
On the other hand, evaluative questions that ask a novice to 
evaluate whether she understands material may not be par- 
ticularly helpful pedagogically because novices often cannot 
identify what they do not understand, or may be hesitant 
to speak up even if they are aware that they are confused. 
Nonetheless these questions occurred regularly in our corpus 
with experienced (though not expert) human tutors. We in- 
vestigate whether students’ affective response to these types 
of tutor dialogue moves is significantly predictive of student 
engagement and frustration as reported at the end of the 
session. 


3.2 Facial Expression Features 


Student facial expressions were automatically extracted 
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ASSIGNMENT 


Now your game needs to get and store the player's latest choice (3 or 4). But remember, your program must not “forget” the 


So I'm thinking | should make a 
choicetwoa and a choicetwob here 


player's previous choice (1 or 2), because the newest scene you output will depend on both choices. 


You can copy a value from one variable to another. When you do this, the value stored in the variable on the right gets stored in 


the variable on the left too. Here's an example. 


leftVar = rightVar; 


(00:11:47) ca 


Like those are going to be my new 
variables or something. 


Task 4 of 9: Without writing any code, make a plan with your tutor to store the player's latest choice. (Hint: you will need B® (00:12:02) 


another variable.) 


OD ) eut copy paste 


ion = “textastic"; 

-println (namelocation) ; 

layername; 

ayerInput; 

8 playerInput = new Scanner (System.in); 

9 System.out.println ("Enter your name here:"); 
0 playername = playerInput.nextLine(); 


:" + playername) ; 
in a field of corn."); 
rth, 2. Sit down."); 


4 System.out.pr: 
5 int choiceone; 


6 choiceone = playerInput.nextint (); 


Okay, so what would you store in those 
two new variables? 


(00:12:32) f& 
= + choicetwoa would have the options if 
you had entered 1 for choiceone and 
choicetwob would have the options if 

you had entered 2 for choiceone 


I (00:13:03) 

Hmm, that's not bad, but you could 
store the player's second choice in just 
one new variable, regardless of what 
thre first choice was, right? 


C3 (00:13:16) 
Let's say that teh player chose 1 first 


7 if(cheiceone == 1) { System.out.println ("Looking north you see a farmhouse."); System.out.4V¥ 


<4 


COMPILE RUN 


COMPILE OUTPUT RUN OUTPUT 


Compiled Successfully! 


md (00:13:36) 


Restore Code from Latest Compile | They still either choose 3 or 4 in the 


second choice 
-| + 


Figure 1: The web-based tutorial interface for Java programming. 


using a state-of-the-art facial expression recognition tool- 
box, FACET (commercial software preceded by a research 
version known as the Computer Emotion Recognition Tool- 
box, CERT) [26]. FACET tracks the frame-by-frame pres- 
ence of several facial action units according to the Facial 
Action Coding Scheme [25]. These action units include 
movements such as AU6 CHEEK RAISER, AU12 Lip Cor- 
NER PULLER, AU24 Lip PREssoR, and AU26 JAw Drop 
(see Figures 5 and 6 for illustration). For each facial action 
unit, the FACET software suggests an Evidence measure, 
indicating the chance that the target expression is present. 
This Evidence measure is on a scale where negative values 
represent evidence of the absence of a facial expression and 
positive values indicate evidence of the presence of one. The 
more positive the measure, the more confident FACET is 
that the feature is present. 


3.3 Gesture Features 


The Kinect depth camera also tracked hand-to-face ges- 
tures made by the student during the tutoring session. An 
algorithm developed to detect such gestures was developed 
to recognize one or two hands touching the lower face. In 
order to do this, the algorithm relies on surface propagation 
from the center of the head, identifying round (i.e. a normal 
head shape) or oblong shapes (i.e., shapes extending beyond 
the normal head shape) based on distances from the center 
of the head. This gesture detection algorithm was previously 
found to be 92.6% accurate when compared against manual 
labels [14]. 


4. ANALYSIS 


The present analysis focuses on the affective response of 
a student, as observed by multimodal traces of face and 


gesture, after tutor inference questions and evaluative ques- 
tions. We hypothesize that multimodal features after these 
tutor questions can predict student engagement and frus- 
tration. In particular, we examine three seconds after each 
tutor dialogue move (a manually-determined interval). The 
multimodal response of the student was characterized using 
the following categories of features, all of which were pro- 
vided to the predictive models. However, note that only the 
first two of these categories of features (shown in bold below) 
appear significantly predictive within the models. 


1. Average evidence measure for each of the facial 
expression action units during the interval (19 
features) 


2. Percentage of the interval in which a one-hand- 
to-face or two-hands-to-face gesture was observed 
(2 features) 


3. Number of skin conductance responses identified dur- 
ing the interval as measured by a skin conductance 
response bracelet (1 feature) 


4, Average student distance from the workstation during 
the interval (1 feature) 


5. Average difference between the highest and lowest points 
of the student’s body from the workstation during the 
interval, indicating leaning (1 feature) 


We calculated the average value of each multimodal fea- 
ture listed in the categories above across each tutoring ses- 
sion. For each feature, we computed its conditional proba- 
bility of occurring after the tutor moves of inference question 
or evaluative question. We also provided the model with the 
overall occurrence of that feature across the entire tutoring 
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Figure 2: Multimodal instrumented tutoring session, including a Kinect depth camera to detect posture and gesture, a webcam 
to detect facial expression changes, and a skin conductance bracelet to detect electrodermal activity. 


Figure 3: Dialogue excerpt illustrating a tutor inference 
question in context. 


Student compiles the program, encounters an error. 
STUDENT Oh. 


TuToR So how can we fix this? 


STUDENT Hmm. 


STUDENT Switch the prompt line with the response 
line? 


Tutor Okay, try it. 


session in order to control for the influence of the feature 
overall (rather than only after the tutor moves of interest). 
Specifically, the features conditional on tutor moves were av- 
erages of the form Avg(Feature|TutorQ) for each student 
that completed the session. The session-wide average of each 
feature, Avg(Feature) were also provided to the model for 
each multimodal feature in all of the categories above. 

Standardization was performed on each feature by sub- 
tracting the mean and dividing by the standard deviation, so 
that the regression coefficients would be more interpretable. 
The standardized features were provided to a stepwise re- 
gression modeling procedure optimizing for the leave-one- 
student-out cross-validated R? value (the coefficient of deter- 
mination), while at the same time requiring a strict p < 0.05 
cut-off value after Bonferroni correction on significance val- 
ues. 


5. RESULTS AND DISCUSSION 


For both types of tutor question, evaluative and inference, 


Figure 4: Dialogue excerpt illustrating a tutor evaluative 
question in context. 


STUDENT Do I need to set the player input before line 
13? 
TuTorR The while tests that [variable]. You need to 
be sure it enters the loop at least once. 
TUTOR Good. 
TuToR Does that make sense? 
STUDENT Yeah. 
STUDENT But what happens if I don’t enter 1 or 2? 


a predictive model was built to predict student frustration 
and student engagement, resulting in a potential four mod- 
els. Three of the four models uncovered significant predic- 
tive relationships. The following subsections detail models 
predicting frustration after tutor inference and evaluative 
questions, and a model predicting engagement after tutor 
evaluative questions. 


5.1 Frustration 


The results suggest that student facial expressions are sig- 
nificantly predictive of self-reported end-of-session frustra- 
tion. The predictive model for student frustration based 
on tutor evaluative questions includes two features, both of 
which are facial action units occurring in the three-second 
interval following the tutor evaluative question (Table 1). 

Two facial action unit features after tutor evaluative ques- 


1The models reported in this paper were built as a part 
of a larger exploratory analysis. As a result, the p-values 
reported have been modified by a Bonferroni correction 
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Table 1: Predictive model for standardized end-of-session 
frustration after tutor evaluative questions (TutorQE).' 


Table 2: Predictive model for standardized end-of-session 
frustration after tutor inference questions.+ 


Leave-One-Out Cross-Validated R? = 0.3235 


Leave-One-Out Cross-Validated R? = 0.3392 


Frustration = | R? | p Frustration = | R? | Dp | 

—0.7039 * AU12 after TUTORQE | 0.0764 | 0.014 +0.5660 * AU6 after TuTORIQ | 0.2893 | 0.022 | 
—0.6279 * AU28 after TUTORQE | 0.2471 | 0.030 +0.3635 * AU20 | 0.0499 | 0.019 | 
—0.1635 (Intercept) | 1.000 —0.0174 (Intercept) | 1.000 | 
| 


tions are significantly predictive of student frustration. Higher 


intensity levels of of AU12 Lip CORNER PULLER (Figure 5b) 
following a tutor evaluative question are negatively indica- 
tive of frustration, as is the presence of AU28 Lip Suck 
(Figure 5d). AU12 is associated with smiling, which is typ- 
ically not associated with frustration although on occasion, 
the two can go hand in hand [21]. 

AU 28 is a type of lower face movement sometimes as- 
sociated with fidgeting, and this type of motion may be a 
*self-manipulator” that is part of emotion regulation. It is 
possible that students engaged in this challenging learning 
task may exhibit this movement to alleviate negative emo- 
tions related to frustration, resulting in lower self-reported 
frustration at the end of the session. When students are 
faced with a question that asks them to evaluate whether 
they understand the material being tutored, these facial ex- 
pressions may both reflect the presence of emotion regulation 
that could mitigate the students’ overall feeling of frustra- 
tion. 

The next model examines student responses to tutor infer- 
ence questions. In contrast to evaluative questions, inference 
questions ask students to bring pieces of knowledge together 
to infer the answer to a question and then to express a sub- 
stantive answer. Two facial action unit features exhibited 
following these questions appear as significantly predictive 
of student frustration. The model shows that AU6 CHEEK 
RAISER (Figure 5a) after tutor inference questions is posi- 
tively predictive of frustration, as is the overall session oc- 
currence of AU20 Lip STRETCHER (Figure 5c). The model 
is displayed in Table 2. 

Interestingly, AU6 has been related to pain expressions in 
the literature on pain detection [28]. When asked to answer 
an inference question, it is possible that students exhibited 
a ”pained” expression that coincides with frustration. The 
expression of AU20 has been observed to coincide with mo- 
ments of embarrassment or awkwardness [24], when people 
were embarrassed or amused in the period after doing di- 
rected facial actions (the technique used to develop images 
for the Facial Action Coding System). AU20 only occurred 
among embarrassed participants in that study. When faced 
with a tutor inference question, this expression may indicate 
that the student is unsure, awkward, or embarrassed, which 
may unsurprisingly be related to frustration. Deeper future 
investigation of subsequent student dialogue moves will help 
elucidate this phenomenon. 


5.2 Engagement 


Next we built models to predict student engagement based 
on affective responses to tutor inference questions and eval- 


p < a/n, where n = 21 is the number of statistical tests 
conducted in the larger analysis, in order to reduce the fam- 
ilywise error rate to a = 0.05. 


(a) AU6 (b) AU12 
CHEEK RAISER + ONE HAND Lip CORNER PULLER 
TO FACE 


(c) AU20 
Lip STRETCHER 


(d) AU28 
Lip Suck 


Figure 5: Sample frames from the student webcam illustrat- 
ing the facial action unit features appearing in the predictive 
models for student frustration, as identified by FACET. 


uative questions. For inference questions, none of the fea- 
tures provided to the model were predictive of engagement. 
However, for affective response to tutor evaluative questions, 
there were seven predictive features, three of which are spe- 
cific to the interval following the event, and four of which 
are session-wide (Table 3). 

The model suggests that facial expression features ac- 
count for most of the variance in predicting student engage- 
ment; however, one session-wide gesture feature was also 
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Table 3: Predictive model for standardized engagement after 
tutor evaluative questions.* 


Engagement = R? Pp 
+0.4422 * ONEHTF 0.1815 | < 0.001 
—0.5989 * AU10 after TUTOREQ 0.1831 | < 0.001 
+0.5770 * AU12 0.2280 | < 0.001 
+0.5097 * AU26 after TUTOREQ 0.0514 | < 0.001 
—0.2941 * AU2 0.1923 | 0.003 
+0.2467 * AU5 0.0295 | 0.002 
+0.1792 * AU24 after TUTOREQ 0.0566 | 0.018 
+0.4100 (Intercept) 1.000 
Leave-One-Out Cross-Validated R? = 0.9224 


selected. The more frequently a student was displaying a 
ONEHANDTOFACE gesture, which may indicate thoughtful 
contemplation, the more engaging the student reported the 
experience at the end of the session. 

Three more session-wide facial expression features were se- 
lected as significantly predictive of student engagement. The 
more intense the expression of AU12 Lip CORNER PULLER 
(Figure 5b) or AU5 Upper Lip RAISER (Figure 6b), the 
more engaged the student. For AU12 which is often asso- 
ciated with smiling, a positive emotion is likely related to 
higher engagement. In this task, AU5 is likely associated 
with the student looking at the screen, possibly indicating 
paying attention and focusing on the task (as opposed to 
the opposite facial movement of blinking or shutting one’s 
eyes). In contrast, AU2 OUTER Brow RAISER (Figure 6a) 
was predictive of lower engagement. This action unit is a 
component of the “fear brow” (AU1+2+4) which has been 
evidenced as a display of anxiety [19]. 

Narrowing down to the context of three seconds after tu- 
tor evaluative questions, three facial expression features were 
significantly correlated with student engagement. The more 
that a student expresses AU26 Jaw Drop (Figure 6e), or the 
more that the student expresses AU24 Lip PRESSOR (Figure 
6d), the more engaged the student reported being at the end 
of the session. Jaw drop is a dynamic action unit that may 
occur when the mouth is closed or already partly open. In 
either case, this action unit may be associated with focus on 
the task, although it could also plausibly be associated with 
a yawn (which we would not expect to coincide with higher 
engagement). With respect to AU24, which is a prototypi- 
cal component of anger, an important interplay of learning 
and affect expression emerges. Some facial movements that 
are part of prototypical displays of negative basic emotions, 
such as anger, appear to be indicative of mental effort dur- 
ing learning, rather than negative affect [31]. From this 
perspective, it makes sense that this AU24 would be related 
to engagement. On the other hand, the more that a student 
expressed AU10 UPPER Lip RAISER (Figure 6c) during this 
interval, the less engagement reported by the student at the 
end of the session. This action unit, which is a component of 
prototypical disgust, is likely to run contrary to engagement. 


6. CONCLUSION 


Tutor dialogue moves in one-on-one human tutoring sig- 
nificantly influence student outcomes, both cognitive and 
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affective. This paper has examined students’ affective re- 
sponse to two types of tutor questions: inference questions 
which require some reasoning to construct an answer, and 
evaluative questions, which ask students to reflect on the 
extent to which they understand the material. The re- 
sults show that immediately after these tutor questions, stu- 
dents’ affective displays—particularly with respect to facial 
expression—are highly predictive of the outcomes of frustra- 
tion and engagement. By detecting these affective displays 
which have been associated in prior studies with emotions 
such as embarrassment, disgust, or happiness, we can begin 
to understand the moment-by-moment affective processes 
that influence learning through tutorial dialogue, and relate 
those fine-grained events to overall outcomes. 

While these facial movements have been associated with 
prototypical emotion displays in the literature, it is impor- 
tant to further contextualize the moments in which these 
expressions appear during tutoring. For instance, action 
units typically associated with anger are likely indicators of 
mental effort during learning. Similarly, an action unit asso- 
ciated with disgust (e.g., AU10) may be related to students’ 
appraisal of the tutor’s question in the moment. Further re- 
search seek to ground these interpretations more extensively 
across salient moments of tutoring. 

There are several additional directions for future work. 
Detecting important moments during tutoring is an open 
area of investigation, with evidence suggesting that moment- 
by-moment affect may be related to distal outcomes [36, 1]. 
In future work, it will be important to expand our under- 
standing of the identified non-verbal predictors for frustra- 
tion and engagement more deeply. We must consider a wider 
variety of contexts, and explore different widths of time af- 
ter tutorial events to examine affective responses with longer 
(or shorter) times to manifest. It is hoped that this line of 
investigation will lead to richer affect models for tutorial 
dialogue. 
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Figure 6: Sample frames from the student webcam illustrat- 
ing the facial action unit features appearing in the predic- 
tive model for student engagement, as identified by FACET. 
Note that AU12 Lip CorRNER PULLER (Figure 5b) also ap- 
pears in these models. 
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